Skip to main content

View Post [edit]

Poster: lawnchair Date: Apr 23, 2015 9:07am
Forum: web Subject: How long until Wayback checks robots.txt again?

Didn't quite see this as an FAQ. I have a site I'd like archived (and like to see the old material). It was inadvertently robots.txt-ed into oblivion. That's been fixed, but IA hasn't checked it since the new robots.txt. How long does Wayback cache that 'disallow'?

Reply [edit]

Poster: Jeff Kaplan Date: Apr 23, 2015 10:21am
Forum: web Subject: Re: How long until Wayback checks robots.txt again?

should be about 24 hours after a change